Explaining Portuguese exam results using CP and PDP profiles¶
The goal of this homework was to explain the results of students from their lifestyles and life situations. Similarly to the previous homework, I use the Portuguese dataset only. For explanations I use cp and pdp from the dalex module. I found that in the dataset that I am using, most datapoints have similar CP profiles (modulo translation) for the majority of variables. From this follows, that PDP profiles don't differ much from CP profiles in terms of shape. There isn't much difference between PDP profiles for XGBoost and Random Forest Regressor - which are the two models that I am explaining.
Task 1¶
$f(x_1, x_2)=(x_1 + x_2)^{2}$
Using basic property of expected value: $g_{PD}^1(z)=\mathbb{E}_{x_2}[z^2 + 2zx_2 + x_2^2]=z^2 + 2z\mathbb{E}_{x_2}[x_2] + {E}_{x_2}[x_2^2]=z^2 + 2z*0 + \frac{1}{2}(\frac{1}{3}- (-\frac{1}{3})) = z^2 +\frac{1}{3}$
Task 2¶
Subtask 2¶
For majority of variables CP profiles are constant or nearly constant. The plots for them are included in the appendix. The plots, that I found most interesting are presented below:
From those plots we can say, that those variables are most likely important in determining the student's score. We can suspect, that there are some monotonic dependencies between those variables and test results, but there are also those examples in which the result change is negligeble. To explain the meaning of those variable names - age is student's age, failures is the number of times the student failed the exam previously, studytime is how much a student studies (scale 1-5), higher is one when a student plans to pursue higher education, Dalc is how much alcohol does a student consume during the weekdays (scale 1-5) and traveltime is how much time does it take the student to get to the school (scale 1-5).
From our (or at least my) priors based on the lifelong experience:
- Older students should have better results, except for those who are really old
- Failing an exam before should be connected to worse test performance
- The amount of alcohol consumed during weekdays should be correlated with worse test performance.
- Bigger time travel to school should be correlated with worse test performance.
- Students who spent more time studying should have better test results.
- Student who seek higher education should have better test results.
Those priors were also confirmed in at least some of previous homeworks.
Based only on the CP profiles:
- Older students have better results, except for those who are really old
- If a student has failed an exam at least once before, their expected score falls down significantly
- The amount of alcohol consumed during weekdays seems to be connected with worsening test performance, but in the case of at least one student, increasing it is 'predicted' to lead to improved test performance. Hence we aren't sure about the influence of alcohol consumption on test performance.
- Small increase in the time travel to school takes seems to be irrelevant, but a bigger one leads to decreased test results. We can't be sure if increase from 2 to 3 matters.
- Students who spent more time studying have better test results (but then in the case of at least two students out of five, the influence is almost non-existant), so we can't be sure if the influence is real.
- Student who seek higher education seem to have better results, but at least for two students it seems irrelevant, so we can't really be sure.
Subtask 3¶
As we can observe in the subtask 2, for dalc, there is a student, for whom a decreased alcohol consumption in a week day would decrease their test results. I would guess, that this is either caused by the fact, that there aren't many students, who don't drink alcohol and have poor test results. It is also possible, that this is caused by some interesting correlations in the data.
Subtask 4¶
In comparison to CP profiles, now more clear patterns can be observed which mostly agree with my priors.
- Older students have better results, except for those who are really old
- If a student has failed an exam at least once before, their expected score falls down significantly
- The amount of alcohol consumed during weekdays seems to be connected with worsening test performance.
- Increase in the time travel to school takes seems to be irrelevant, unless the student falls into the bin of students for whom the travel takes the longest
- Students who spent more time studying have better test results.
- Student who seek higher education have better results.
Subtask 5¶
For most of the variables, the profiles for Random Forest Regressor and XGBRegressor are fairly similar. There are some exceptions - age and travel time seem to matter more in the case of XGBRegressor and failures in the case of Random forest regressor.
Appendix¶
Contains the Python code used to create the above raport
Dataset¶
This dataset contains the data about the scores of students of two different school in Portugal in math and Portuguese. The goal is to predict the final score of a student, using information about their lifestyle and background. I am going to focus only on predicting G3 - according to the description of the problem G1 and G2 are highly correlated with G3.
Imports¶
We need to install the necessary modules.
!pip install dalex
And now import everything that is necessary.
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import dalex as dx
import random
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestRegressor
from xgboost import XGBRegressor
from sklearn.metrics import mean_squared_error
from sklearn.feature_selection import RFECV
Set the seeds for reproducibility.
np.random.seed(10)
random.seed(10)
Data preparation¶
We will start by reading the data.
portuguese_df = pd.read_csv('Portuguese.csv')
Split into features and targets. The same as in the previous homework, we focus on predicting the final result only.
port_x = portuguese_df.drop(['G3', 'G2', 'G1'], axis = 1)
port_y = portuguese_df[['G3']]
Categorical variables need to be encoded.
def encode_categorical(df):
df = df.copy()
df = df.replace({'GP': 0, 'MS': 1})
df = df.replace({'F': 0, 'M': 1})
df = df.replace({'U': 0, 'R': 1})
df = df.replace({'LE3': 0, 'GT3': 1})
df = df.replace({'A': 0, 'T': 1})
df = df.replace({'no': 0, 'yes': 1})
onehot_features = ['Mjob', 'Fjob', 'reason', 'guardian']
df = pd.concat([df, pd.get_dummies(df[onehot_features])], axis=1)
df = df.drop(onehot_features, axis=1)
return df
port_x = encode_categorical(port_x)
There is little data available. There is much more data about results from portuguese than from math. For that reason, we are going to train our models using both datasets at once and then test them separately.
Homework implementation¶
Below the code used for solving each of the required homework tasks is contained. Only XGBoost is used with the exception of the subtask 7.
Subtask 0¶
We train two models XGBoost and random forest model. We train on the whole dataset, since there isn't much data and it shouldn't matter for the explanations.
models = {
'Boosting': XGBRegressor(objective='reg:squarederror'),
'Forest': RandomForestRegressor()
}
for name, model in models.items():
model.fit(port_x, port_y.values.ravel())
Subtask 1¶
We sample a few random observations from the dataset and calculate the predictions.
n_samples = 5
samples = port_x.sample(n_samples)
models["Boosting"].predict(samples)
Subtask 2¶
Calculating what-if explanations of predictions using CP profiles.
explainer = dx.Explainer(models['Boosting'], port_x, port_y)
cp = explainer.predict_profile(new_observation=samples)
cp.plot(variables=["Dalc", "Walc"])
cp.plot(variables=["failures"])
cp.plot(variables=['school', 'sex'])
cp.plot(variables=['age', 'address'])
cp.plot(variables=['famsize', 'Pstatus'])
cp.plot(variables=['Medu', 'Fedu'])
cp.plot(variables=['traveltime', 'studytime'])
cp.plot(variables=['failures', 'schoolsup'])
cp.plot(variables=['paid', 'famsup'])
cp.plot(variables=['activities', 'nursery'])
cp.plot(variables=['higher', 'internet'])
cp.plot(variables=['romantic', 'famrel'])
cp.plot(variables=['freetime', 'goout'])
cp.plot(variables=['health', 'absences'])
Subtask 3¶
Examples, where there are ones for which the prediction is increasing and ones for which it is decreasing for the same variable.
cp.plot(variables=['Dalc'])
Subtask 4¶
Comparing pdp with cp results.
pdp = explainer.model_profile()
pdp.plot(variables=["age", "failures", "Dalc", "traveltime", "studytime", "higher"], title="PDP Profiles")
cp.plot(variables=["age", "failures", "Dalc", "traveltime", "studytime", "higher"])
Subtask 5¶
Comparison of pdp for two different models.
explainer_forest = dx.Explainer(models['Forest'], port_x, port_y)
pdp_forest = explainer_forest.model_profile()
pdp_forest.plot(pdp, variables = ["age", "failures", "Dalc", "studytime", "traveltime", "famrel", "higher"])